Image display method, image coding apparatus, and image decoding apparatus

ABSTRACT

An image coding apparatus is provided which satisfies various levels of demands on image distribution, both from image providers and from users. The image coding apparatus includes a coding block which codes predetermined image data. A separation unit separates the coded image data into basic data for reproducing contents of the coded image data as a visible image, and complementary data for complementing the basic data, so that the two pieces of data are distributed on different occasions. An adding unit adds information for independent copyright control to at least either one of the basic data and the complementary data.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to an image display method, an image codingapparatus, and an image decoding apparatus which are intended fordistribution of image data capable of being separated into a pluralityof elements.

2. Description of the Related Art

In recent years, the prevalence of such infrastructures as DVD media,digital broadcasting, and on-demand network communications has producedwidespread use of digital contents. Since digital contents cause nodegradation in quality even after duplication, copyright management isof high importance. Demands for legitimate protection of copyright ondigital contents and other copyrighted materials are expected to grow inthe future.

The number of digital contents accessible over the Internet isincreasing year by year, and the number of Internet users is as well.This has resulted in an increasing amount of traffic on the Internet. Inthis respect, Japanese Patent Laid-Open Publication No. Hei 9-46677discloses an image transmission apparatus which switches among a modefor transmitting only the I frames of MPEG data, a mode for transmittingthe I frames and P frames of the same, and a mode for transmitting the Iframes, P frames, and B frames of the same depending on the traffic on atransmission line.

Even with such a technique, however, various levels of demands thatimage providers and users have cannot be fully satisfied. For example,image providers aim for copyright protection and, at the same time, wishto have their contents shown to a large number of users. Meanwhile,users have various orientations ranging from high-end toprice-sensitive.

SUMMARY OF THE INVENTION

The present invention has been achieved in view of the foregoingcircumstances. It is thus an object of the present invention to providean image display method, an image coding apparatus, and an imagedecoding apparatus which are capable of flexible provision of images,beneficial to both image providers and users who use the images.

To solve the foregoing problems, an image distribution method accordingto one of the aspects of the present invention comprises: separatingpredetermined image data into basic data for reproducing contents of thepredetermined image data as a visible image, and complementary data forcomplementing the basic data for the sake of improved image quality;first distributing the basic data; and second distributing thecomplementary data on an occasion different from the distribution of thebasic data. The “second distributing” may include distributing thecomplementary data in response to a request from a destination of thedistribution. The “image quality” may include smoothness of motion of amoving image.

According to this aspect, the separation of the image data can increasethe variety of modes in which images are provided to users. This allowsflexible provision of images which is beneficial to both image providersand users.

The method may further comprise managing copyright on the basic data andthe complementary data independently. According to this aspect, it ispossible to exercise flexible copyright management, for example, suchthat either one data is put under a copyright control like storageprohibition while the other is allowed for free use.

When separated, the basic data may include at least either one of adirect-current component and a low frequency component of the imagedata. According to this aspect, the image data is separated spatially,with at least either one of its direct-current component and lowfrequency component as the basic data. This makes it possible togenerate basic data that has comprehensible contents.

When separated, the basic data may include an intra-frame coded frameout of a plurality of frames constituting the image data. Thecomplementary data may include a frame that is expressed as a differencefrom another frame. According to this aspect, the image data isseparated temporally, with a frame(s) reconstructable independently asthe basic data. This makes it possible to generate basic data that hascomprehensible contents.

Another aspect of the present invention is an image coding apparatus.This apparatus comprises: a coding unit which codes predetermined imagedata; and a separation unit which separates the coded image data intobasic data for reproducing contents of the coded image data as a visibleimage, and complementary data for complementing the basic data for thesake of improved image quality.

According to this aspect, the separation of image data can increase thevariety of modes in which images are provided to users. This allowsflexible provision of images which is beneficial to both image providersand users.

The apparatus may further comprise an adding unit which adds informationintended for independent copyright management to at least either one ofthe basic data and the complementary data. The “adding unit” may recordthe information when generating a stream of the coded image data, orrecord the information during postprocessing after the coding.

The separation unit may separate the image data so that the basic dataincludes at least either one of a direct-current component and a lowfrequency component of the image data. The basic data may otherwiseinclude an intra-frame coded frame out of a plurality of framesconstituting the image data.

Yet another aspect of the present invention is an image decodingapparatus. This apparatus comprises: a recording unit which records inadvance basic data for reproducing contents of predetermined image dataas a visible image; an acquisition unit which acquires complementarydata for complementing the basic data for the sake of improved imagequality; an assembling unit which assembles the basic data and thecomplementary data; and a decoding unit which decodes the assembledimage data. The “decoding unit” may decode the basic data.

According to this aspect, the provision of the function for assemblingseparated pieces of image data makes a mechanism for flexible provisionof images, beneficial to both image providers and users, feasible.

The apparatus may further comprise a copyright management unit whichrefers to information intended for independent copyright management andexercises copyright management, the information being added to at leasteither one of the basic data and the complementary data. According tothis aspect, a mechanism by which image providers exercise flexiblecopyright management becomes feasible.

The apparatus may further comprise a data request unit which requestsdata for complementing the basic data from a source of distributionafter the basic data is acquired, depending on processing capability ofthe apparatus. According to this aspect, it is possible to suppress thedistribution of useless complementary data that exceeds the processingcapability of the apparatus.

Yet another aspect of the present invention is an image displayapparatus. This apparatus comprises: an acquisition unit which acquiresbasic data for reproducing contents of predetermined image data as avisible image and complementary data for complementing the basic datafor the sake of improved image quality on respective differentoccasions, the predetermined image data being separated into the basicdata and the complementary data; and a display unit which displays imagedata assembled from the basic data and the complementary data. The“display unit” may decode and display the basic data.

According to this aspect, the provision of the function for assemblingseparated pieces of image data makes a mechanism for flexible provisionof images, beneficial to both image providers and users, feasible.

The apparatus may further comprise a copyright management unit whichrefers to information intended for independent copyright management andexercises copyright management, the information being added to at leasteither one of the basic data and the complementary data. The apparatusmay further comprise a data request unit which requests data forcomplementing the basic data from a source of distribution after thebasic data is acquired, depending on processing capability of theapparatus.

Incidentally, any combinations of the foregoing components, and thecomponents and expressions of the present invention mutually replacedwith methods, apparatuses, systems, recording medium, programs, and thelike are also intended to constitute applicable aspects of the presentinvention.

Moreover, this summary of the invention does not necessarily describeall necessary features so that the invention may also be sub-combinationof these described features.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described by way of examples only, withreference to the accompanying drawings which are meant to be exemplary,not limiting and wherein like elements are numbered alike in severalFigures in which:

FIG. 1 is a diagram showing the procedure of image coding processing;

FIG. 2 is a diagram showing the configuration of an image codingapparatus according to embodiment 1;

FIG. 3 is a diagram showing the structures of coded image data beforeand after spatial separation;

FIG. 4 is a diagram showing an example of a copyright level managementtable;

FIG. 5 is a block diagram showing the configuration of an image decodingapparatus according to embodiment 2;

FIG. 6 is a diagram showing the structures of coded image data beforeand after temporal separation;

FIG. 7 is a block diagram of an image coding apparatus according toembodiment 5;

FIG. 8 is a block diagram of an image decoding apparatus according toembodiment 6;

FIG. 9 is a diagram showing an example where resolution is lowered withtime;

FIG. 10 is a diagram showing an example where image quality is loweredwith time;

FIG. 11 is a diagram showing an example where the display area isnarrowed down with time;

FIG. 12 is a diagram showing an example where an image is scaled downwith time;

FIG. 13 is a diagram showing an example where the frame rate is reducedwith time;

FIG. 14 is a block diagram of an image coding apparatus according toembodiment 7;

FIG. 15 is a diagram showing the procedure of image coding processing;

FIG. 16 is a diagram showing an example of the data structure of a codedimage data stream;

FIG. 17 is a diagram showing an example of copyright level information;

FIG. 18 is a diagram showing the configuration of an image processingapparatus according to embodiment 8 of the present invention;

FIG. 19 is a diagram showing the data structure of a memory unitaccording to embodiment 8;

FIG. 20 is a diagram showing a screen on which images having the samecontents but different image qualities are displayed in a multi-windowfashion; and

FIG. 21 is a diagram showing a screen on which images having the samecontents but different image qualities, including one having noisesuperimposed thereon, are displayed in a multi-window fashion.

DETAILED DESCRIPTION OF THE INVENTION Embodiment 1

The present invention achieves an image provision system which isappealing to both image providers who provide still and/or moving imagesand users who use the images. Embodiment 1 will deal with an imagecoding apparatus 100 which separates image data spatially. For example,the image coding apparatus 100 separates direct-current components andthe remaining frequency components as basic data and complementary data,respectively. Then, the two pieces of data are distributed separately orsubjected to copyright management independently.

Embodiment 1 can use technologies for generating still images or movingimages of different image qualities from a single coded image datastream. By way of example, a method for coding a moving image by usingMotion-JPEG 2000 scheme will be described briefly with reference toFIG. 1. A not-shown image coding apparatus generates a coded data streamof a moving image by coding individual frames of the moving imagecontinuously frame by frame. At the beginning of the coding process, anoriginal image OI 102 corresponding to a single frame of the movingimage is read into a frame buffer. The original image OI read into theframe buffer is hierarchized by a wavelet transformer.

The JPEG-2000 wavelet transformer uses a Daubechies filter. This filterfunctions as a high-pass filter and a low-pass filter both in x and ydirections of an image simultaneously, thereby dividing the single imageinto four frequency subbands. The subbands consist of an LL subband, anHL subband, an LH subband, and an HH subband. The LL subband containslow frequency components both in the x and y directions. The HL subbandand the LH subband contain low frequency components in either one of thex and y directions and high frequency components in the other direction.The HH subband contains high frequency components both in the x and ydirections. This filter also has the function of reducing the number ofpixels by half both in the x and y directions. That is, each subband hasas many pixels as a half those of the unprocessed image both in thevertical and horizontal directions. A single application of thefiltering thus produces subband images having a resolution, i.e., imagesize of ¼. As employed in this specification, an image obtained byapplying a single wavelet transform to the original image OI will bereferred to as a first-level image WI1. Hereinafter, an nth-level imagewill be referred to as WIn in accordance with the number of wavelettransforms applied thereto.

As schematically shown in FIG. 1, the first-level image WI1 104 has foursubbands LL1, HL1, LH1, and HH1. A wavelet transform is applied to thefirst-level image WI1 104, whereby a second-level image WI2 106 isgenerated. The second and subsequent wavelet transforms will be appliedto only the LL subband components of the images in the respectiveprevious levels. Consequently, in the second-level image WI2 106, theLL1 subband of the first-level image WI1 is decomposed into foursubbands LL2, HL2, LH2, and HH2. The wavelet transformer performs thisfiltering a predetermined number of times, and outputs the wavelettransform coefficients of the respective subbands. The image codingapparatus then performs quantization and other processing, and finallyoutputs coded image data CI.

For ease of explanation, the image coding apparatus in this exampleshall apply three wavelet transforms to the original image OI. Suppose,for example, that the original image OI 102 has 1440×960 pixels. Then,the LL1 subband of the first-level image WI1 104 has a size of 720×480,the LL2 subband of the second-level image WI2 106 a size of 360×240, andthe LL3 subband of the third-level image WI3 108 a size of 180×120.

As far as the hierarchical images are concerned, it should be noted thatthe low frequency components of the original image OI gather around theupper left in FIG. 1. In the case of FIG. 1, the LL3 subband which fallson the upper left corner of the third-level image WI3 is the lowest infrequency. Conversely, the most basic properties of the original imageOI can be reproduced as long as this LL3 subband is accessible.

Aside from Motion-JPEG 2000, the coded data stream may be of, e.g., SVC(Scalable Video Codec) which provides a single stream including both ahigh quality HD stream and a low quality SD stream in combination.Motion-JPEG is also applicable. Among JPEG schemes are ones in whichframes are transmitted in ascending order of the terms of theircoefficients obtained by discrete cosine transform. Here, image qualitycan be selected depending on up to what order of terms the coefficientsare used for decoding. According to these specifications, spatialresolution can be coded hierarchically.

The foregoing example has been dealt with the case where thehierarchization is achieved by the coding of frequency division type.Nevertheless, coding of improved approximation accuracy type may be usedfor hierarchization. More specifically, the number of higher-order bitsof DCT (Discrete Cosine Transform) coefficients or wavelet coefficientsto be decoded can be adjusted to achieve decoding in different imagequalities. In MPEG-2, not only the foregoing spatial resolution but alsotime resolution can be coded hierarchically by adjusting the number offrames.

FIG. 2 shows the configuration of the image coding apparatus 100according to embodiment 1. In terms of hardware, this configuration canbe achieved by an arbitrary computer CPU, a memory, and other LSIs. Interms of software, it can be achieved by a program which is loaded on amemory and has decoding functions. The functional blocks shown here areachieved by the cooperation of these. It will thus be understood bythose skilled in the art that these functional blocks may be achieved invarious forms including hardware alone, software alone, and combinationsof these.

The image coding apparatus 100 includes a coding block 8, a separationunit 19, a stream generation unit 20, a copyright level management table22, and an adding unit 24. The coding block 8 includes a wavelettransform unit 10, a frame buffer 12, a quantization unit 14, a bitplane coding unit 16, and an arithmetic coding unit 18.

Initially, an original image OI is read into the frame buffer 12. Thewavelet transform unit 10 reads the original image OI from the framebuffer 12, and transforms the image by a wavelet transform recursively.As described above, the JPEG-2000 wavelet transform uses a Daubechiesfilter. This filter functions as a high-pass filter and a low-passfilter both in vertical and horizontal directions of the image, therebydividing the single image into four frequency subbands. Each subband hasas many pixels as ½ those of the unprocessed image both in the verticaland horizontal directions. A single application of the filtering thusproduces subband images having a resolution, i.e., image size of ¼. Thesubband images obtained thus are once stored into the frame buffer 12.

The wavelet transform unit 10 reads the image of the LL subband, or thelowest frequency component out of the resulting subbands, from the framebuffer 12. The wavelet transform unit 10 performs the filtering processagain to divide the subband image into four, i.e., LL, HL, LH, and HHsubbands further, and writes them to the frame buffer 12. The filteringis performed a predetermined number of times, and the LL subbandresulting from the last filtering is acquired as an image the closest toa DC component of the original image OI. The subbands in an identicallevel, i.e., the four subbands obtained through the application offiltering an identical number of times contain high frequency componentsthat increase in order of LL, HL and LH, and HH. These subbands arefollowed by images that contain components of higher frequencies, i.e.,the four subbands obtained in the previous filtering process. Suchvertical and horizontal four-way filtering is applied to the lowestfrequency components repeatedly. As a result, an image having a verticaland horizontal four-way recursive structure, which contains low to highfrequency components in a hierarchical configuration, is obtained in theframe buffer 12.

The quantization unit 14 quantizes the hierarchical image stored in theframe buffer 12 from lower to higher frequency components in successionas needed. The bit plane coding unit 16 renders the waveletcoefficients, quantized and divided into units called code blocks forarithmetic coding, into a bit plane. The arithmetic coding unit 18arithmetically codes this bit plane.

By using the arithmetically coded bit string, the separation unit 19separates each level of bit stream into a plurality of groups. Forexample, the first-level image WI1 104 of FIG. 1 may be separatedbetween a group of the subband LL1 and a group of the other threesubbands HL1, LH1, HH1.

FIG. 3 shows the structures of the coded image data before and after thespatial separation. In each of the frames of such hierarchical codedimage data 50 as discussed above, the data area describes data on adirect current component, a low frequency component, a medium frequencycomponent, and a high frequency component in succession from the top.Thus, it is possible to reproduce images even from the first portions ofthe respective frames alone, whereas the images are low in quality. Theimage improves in quality as the last portions are involved in thereproduction. In FIG. 3, this coded image data 50 is separated intocoded image data 52 on direct-current components and low frequencycomponents, and coded image data 54 on medium frequency components andhigh frequency components. Incidentally, the number of classificationsis not limited to two, but may be three or more. For example, the codedimage data on the medium frequency components and the coded image dataon the high frequency components may be classified separately.

The stream generation unit 20 generates a bit stream from each of thebit strings classified by the separation unit 19. Through theseprocesses, the foregoing original image OI is transformed into aplurality of pieces of coded image data CI which are separatedhierarchically.

These plurality of pieces of coded data CI are provided to users onrespective different occasions by various methods. For example, thecoded image data on direct-current components and low frequencycomponents may be distributed for free. Here, the data may bedistributed as a trial version on DVD-ROM or suchlike media, or may bedownloaded by users over a network. The coded image data on medium andhigh frequency components is provided to the users subsequently. Again,the data may be distributed in media or downloaded from a server. Thiscoded image data may also be provided through streaming distribution soas not to be recorded by the users. This streaming distribution may beprovided to paid users on demand.

According to this technique, users can appreciate images for free aslong as they reproduce the images in low quality, whereas they must payto view the images when reproducing in high quality. Since the separatepieces of coded image data are thus provided to users on differentoccasions by various methods, image providers can make detailedcopyright management with respect to each image quality. This increasesthe range of sales techniques of the image contents. Besides, users canenjoy more options and access image contents of their own tastes.

Hereinafter, description will be given of a technique by which an imageprovider makes more detailed copyright management. The copyright levelmanagement table 22 and the adding unit 24 are used for that purpose.The copyright level management table 22 manages copyright levels to bepermitted to users who use the coded image data CI. FIG. 4 shows anexample of the copyright level management table 22. The applications ofthe coded image data CI are shown on the horizontal axis, and thepermitted image quality levels corresponding to each application on thevertical axis. These permitted image quality levels shall correspond tothe respective pieces of coded image data separated by the separationunit 19. In FIG. 4, reproduction is permitted in all the low, medium,and high image quality levels. Duplication is prohibited in the highimage quality level, and permitted in the low and medium image qualitylevels. Edit is prohibited in the medium and high image quality levels,and permitted in the low image quality level. Here, the editing includessuch operations as removing commercial breaks from coded image data thatis received in a digital television broadcast. Consequently, the datacan be duplicated in the medium image quality, but not when commercialbreaks are removed. Redistribution is prohibited. Redistribution is notpermitted in any of the image quality levels.

The generation of duplication shows duplicable generations and thepermitted image quality levels in those generations. The firstgeneration is duplicable in medium image quality. The duplication of thesecond generation, or the duplication of data from the first generation,can only be permitted in low image quality. None of the fourth andsubsequent generations can be duplicated. The copyright level managementtable 22 manages such a profile as described above user by user. Theindividual items of the profile can be updated through key entry and thelike by administrators when needed.

When the coded image data CI is provided to paid users, the availableapplications and the permitted image quality levels for thoseapplications vary depending on fees paid by the users and price plansselected. The profiles may also be created region by region or time bytime, instead of user by user. For example, time-based profiles can becreated to impose time limits on the availability of contents for users.If user apparatuses are configured to switch profiles after a certainperiod in time, time limits can be set so as to lower the permittedimage quality levels or restrict some uses, instead of making all thecontents unavailable after that period.

The adding unit 24 records copyright level information on thecorresponding profile into header or other areas of the respectivepieces of coded image data CI in consideration of such factors as usersand regions to be provided. The copyright level information may bewritten to header and other areas that can be set by administratorsfreely. In such cases as establishing time limits, copyright levelinformation on a plurality of profiles may be added. Incidentally, thecopyright level information need not be added to coded image data thatcorresponds to image quality levels not subject to copyright management.The copyright level management table 22 and the adding unit 24 may beconfigured as an external independent device. This device can access anetwork to add copyright level information to the coded image data CIthat is being communicated over the network.

As has been described, according to the present invention, it ispossible to make flexible provision of images which is beneficial toboth image providers and users who use the images. Image providers canalso manage copyrights with respect to each of the pieces of coded imagedata separated, thereby achieving detailed copyright management. Theseparation into a plurality of pieces of coded image data includesgenerating coded image data on direct-current components and lowfrequency components from which an image having visible contents can bereproduced. This makes various sales channels and various salespromotion techniques usable. For example, the coded image data can bedistributed as a trial image since it has visible contents. Then, userswho want to view in high image quality request the coded image data onthe remaining frequency components, and the image provider distributesit to the users in response. This allows a saving of the hardwareresources. That is, the amount of traffic on the network can be reducedas compared to the cases where the complete set of coded image dataincluding the high frequency components is distributed to every user.This also makes it possible to save recording capacities on hard disksor the like of the receiving user terminals.

Embodiment 2

Embodiment 2 is an image decoding apparatus 200 which can acquire aplurality of hierarchically separated pieces of coded image data ondifferent occasions, and decode the pieces of coded image dataassembled. Incidentally, the image decoding apparatus 200 also functionsas an image display apparatus if it is equipped with a display unit 38.

FIG. 5 shows the configuration of the image decoding apparatus 200according to embodiment 2. The image decoding apparatus 200 comprises anacquisition unit 32, a recording unit 34, an assembling unit 36, and adecoding block 250. The acquisition unit 32 acquires coded image data CIwhich is separated hierarchically as described in embodiment 1. Theacquisition unit 32 may acquire the data by downloading it over anetwork, or acquire it from a recording medium containing the same.Broadcasting waves may also be used for the acquisition. The acquisitionunit 32 records the coded image data CI acquired into the recording unit34. Incidentally, when the storage of the data is prohibited due tostreaming distribution or the like, the data cannot be recorded on therecording unit 34 and thus is output directly to the decoding block 250.

When the acquired data itself is capable of forming an image havingcomprehensible contents, like coded image data on direct-currentcomponents and low frequency components, the coded image data is outputto the decoding block 250 so that the user can view the image.

The assembling unit 36 separates a plurality of pieces of coded imagedata acquired on different occasions into frames, and assemblescorresponding frames with each other. The assembling unit 36 thenarranges the assembled frames in order, thereby reconstructing theunseparated coded data. For example, the processing is reverse to thatof separating the coded image data shown in FIG. 3 above. In FIG. 3, thecoded image data 52 on direct-current components and low frequencycomponents and the coded image data 54 on medium frequency componentsand high frequency components are both divided into frames. For eachframe, the divided elements are then reconstructed in order of adirect-current component, a low frequency component, a medium frequencycomponent, and a high frequency component from the top of the data area.Finally, the reconstructed frames are connected in order. Incidentally,original coded image data can also be reconstructed from three or morepieces of coded image data by the same technique.

The assembling unit 36 acquires the plurality of pieces of coded imagedata to be reconstructed from at least either one of the acquisitionunit 32 and the recording unit 34. For example, coded image data ondirect-current components and low frequency components that ispreviously recorded on the recording unit 34 and coded image data onmedium frequency components and high frequency components that isstreaming-distributed can be assembled. The coded image data assembledis then output to the decoding block 250.

The decoding block 250 comprises a stream analysis unit 252, anarithmetic decoding unit 254, a bit plane decoding unit 256, an inversequantization unit 258, an inverse wavelet transform unit 260, and aframe buffer 262. The stream analysis unit 252 receives coded image dataCI from the assembling unit 36 or the recording unit 34, and analyzesthe data stream. When copyright level information is included in theheader or the like of the coded image data CI, the stream analysis unit252 detects and passes it to a copyright management unit 40. Thearithmetic decoding unit 254 applies arithmetic decoding to a datastring to be decoded which is obtained by the analysis. The bit planedecoding unit 256 decodes the data resulting from the arithmeticdecoding into a bit plane with respect to each color component. Theinverse quantization unit 258 inversely quantizes the quantized datadecoded. The inverse wavelet transform unit 260 applies inverse wavelettransforms to the nth-level image WIn resulting from the inversequantization, by using the frame buffer 262. Each time an inversewavelet transform is applied to the coded image data CI, an image ofhigher level is obtained. Decoding up to the topmost level producesdecoded image data DI.

The display unit 38 displays the decoded image data DI decoded by thedecoding block 250. In FIG. 5, the display unit 38 is shown inside theimage decoding apparatus 200, whereas it may be installed outside theapparatus 200. The copyright management unit 40 manages the copyright onthe coded image data CI for use in this apparatus 200. What is managedis the copyright level information detected by the stream analysis unit252, or copyright level information acquired through sessions with sserver and the like of the contents provider. For example, when a useruses the coded image data CI, the copyright management unit 40 mayidentify the permitted image quality level corresponding to thatapplication. According to the image quality level, the copyrightmanagement unit 40 then instructs the inverse wavelet transform unit 260of the number of times for the inverse wavelet transform to be applied.In such cases, coded image data for achieving high image quality, evenif included, cannot be reproduced or used with that image quality.

When the recording unit 34 previously contains some of the unseparatedcoded image data, the data request unit 42 requests the distribution ofthe rest of the coded image data to the server or the like of thecontents provider. For example, if the coded image data ondirect-current components and low frequency components is stored inadvance, the complementary coded image data on medium frequencycomponents and high frequency components may be requested. This datarequest may be made in response to user operations or based on thespecifications of this image decoding apparatus 200.

In the cases of user operations, the complementary coded image data maybe charged for. Considering the tradeoff between desired image qualitiesand fees, users can request the coded image data on medium frequencycomponents and high frequency components at relatively high price, orrequest the coded image data on medium frequency components atrelatively low price.

A capability detection unit 44 detects the processing capabilities ofthis image decoding unit 200. For example, it detects the processingcapabilities of the decoding block 250, and the processing capabilitiesof the display unit 38 including resolution. The processing capabilitiesof the decoding block 250 may include computing power and a buffercapacity. Based on the processing capabilities detected, the capabilitydetection unit 44 determines which coded image data to request from theimage provider. For example, the capability detection unit 44 calculatesthe maximum performance of this image decoding apparatus 200, assumescoded image data having highest image quality or resolution within thatrange, and determines coded image data necessary to achieve this. Thecapability detection unit 44 passes to the data request unit 42 theinformation as to the coded image data that is to request from the imageprovider in order to complement the coded image data recorded previouslyon the recording unit 34.

The data request unit 42 passes the information to the image provider.The information need not necessarily be transmitted to the imageprovider in response to user operations, but may be transmittedautomatically over a network. For example, the data request unit 42 maytransmit the information when the coded image data on direct-currentcomponents and low frequency components is acquired. The data requestunit 42 may acquire the maximum performance of the image decodingapparatus 200 from the capability detection unit 44, and transmit it tothe image provider. In this case, the image provider can take theinformation into account when determining the amount of complementarydata on the coded image data, and can thus suppress useless distributionexceeding the processing capabilities of the destination decodingapparatus.

Moreover, when requesting the complementary coded image data in responseto user operations, the data request unit 42 may use the foregoingmaximum performance as the upper limit and issue a complement requestwithin the range conforming to the performance.

As has been described, according to the present invention, it ispossible to provide an image decoding apparatus that contributes to therealization of flexible provision of images which is beneficial to bothimage providers and users who use the images. For example, users canreceive coded image data on direct-current components and low frequencycomponents capable of generating an image having visible contents, viewthe image, and determine if they want to view in higher image quality.For image providers, this technique is securer than in the cases wherethe copyright for high quality reproduction is protected by limiting thedegree of decoding without separating the coded image data, since thenecessary data itself is not delivered to users. Moreover, some usersshould not want to view in high quality, in which case the amount oftraffic on the entire network can be suppressed since the complementarycoded image data need not be transmitted. Requesting the complementarycoded image data from image providers in consideration of the processingcapabilities of the decoding apparatus can also reduce useless datadistribution with a further suppression in traffic.

Embodiment 3

Embodiment 3 will deal with an image coding apparatus 100 whichseparates image data temporally. The image coding apparatus 100separates some of a plurality of frames constituting a moving image asbasic data and the remaining frames as complementary data. Here, some ofa plurality of frames refer to a combination of such frames as extractedfrom every eight frames. Then, the two pieces of data are distributedseparately or subjected to copyright management independently.

The configuration and operation of the image coding apparatus 100according to embodiment 3 are basically the same as in embodiment 1.Hereinafter, description will be given of the differences. Whileembodiment 1 has dealt with the case where the moving image is codedunder the Motion-JPEG 2000 scheme, embodiment 3 will deal with anexample of MPEG-based coding. The MPEG-based coding is a commontechnology, and detailed description thereof will thus be omitted. Todescribe the MPEG-based coding briefly in conjunction with FIG. 2, thecoding block 8 often performs discrete cosine transform instead of thewavelet transform. Another process is also added in which inter-framemotion compensation prediction is performed before coefficienttransformation, so that inter-frame prediction errors are subjected tothe transformation.

The separation unit 19 separates a plurality of frames coded by thecoding block 8 into a plurality of groups. For example, the plurality offrames may be separated into a group of I frames and a group of theothers, or into a group of I frames, a group of P frames, and a group ofB frames. I frames are intra-coded frames, and can form imagesindependently. P frames are ones obtained by coding forward predictionerrors in a series of frames. B frames are ones obtained by codingbidirectional prediction errors in a series of frames.

FIG. 6 shows the structures of coded image data before and after theseparation. Coded image data 60 before the separation includesintra-coded frames 61 and frames coded by using prediction errorsbetween frames. A moving image reproduced from the intra-coded frames 61alone shows jerky image motion like a slide show. It is possible,however, to recognize the contents unless the intra-coded frames 61 areextremely small in proportion.

In FIG. 6, the coded image data 60 is separated into coded image data 62consisting of the intra-coded frames 61 and coded image data 64consisting of the other frames which are coded by using predictionerrors between frames. Incidentally, the number of classifications isnot limited to two, but may be three or more. For example, the foregoinggroup of P frames may be further separated into odd-numbered andeven-numbered ones. It is understood that the group of I frames may alsobe separated.

As has been described, the present embodiment provides the same effectsas those of the foregoing embodiment 1. With temporal separation, stillimages constituting a moving image can be distributed as a trial versionwithout losing image quality. This can provide a high visual impact onusers, thereby promising a high advertising effect.

Embodiment 4

Embodiment 4 is an image decoding apparatus 200 which can acquire aplurality of pieces of coded image data separated into frames ofdifferent coding schemes on different occasions, and decode the piecesof coded image data assembled. Incidentally, the image decodingapparatus also functions as an image display apparatus if it is equippedwith a display unit 38.

The configuration and operation of the image decoding apparatus 200according to embodiment 4 are basically the same as in embodiment 2.Hereinafter, description will be given of the differences. Whileembodiment 2 has dealt with the case of assembling a plurality of piecesof coded image data that are separated spatially, embodiment 4 will dealwith an example of assembling temporally separated ones.

Based on a plurality of pieces of coded image data acquired on differentoccasions, the assembling unit 36 arranges frames in proper order beforethe separation, thereby reconstructing the coded image data before theseparation. For example, the processing is reverse to that of separatingthe coded image data shown in FIG. 6 above. In FIG. 6, the individualframes included in the coded image data 62 consisting of the intra-codedframes 61 and the coded image data 64 consisting of the frames coded byusing prediction errors between frames are rearranged into proper orderbefore the separation, whereby the coded image data 60 before theseparation is reconstructed. Incidentally, original coded image data canalso be reconstructed from three or more pieces of coded image data bythe same technique.

The decoding of MPEG-coded image data is a common technology, anddetailed description thereof will thus be omitted. To describe thedecoding of the coded image data briefly in conjunction with FIG. 5, thedecoding block 250 performs inverse discrete cosine transforms insteadof the inverse wavelet transforms if there are DCT coefficients coded bythe discrete cosine transforms on the coding side. The inverse transformmust also be followed by motion compensation processing based onreference images and prediction errors.

As has been described, the present embodiment provides the same effectsas those of the foregoing embodiment 2. When coded image data istemporally separated and a group of frames thereof that can reconstructimages independently are acquired, it is possible to view still imagesconstituting the moving image without losing image quality.Consequently, users can make a determination whether or not to requestthe complementary data of this moving image, even in terms of quality ofthe image itself. Besides, the time-based assembly of frames can beperformed relatively easily.

Up to this point, the present invention has been described inconjunction with several embodiments thereof. The foregoing embodimentshave been given solely by way of example. It will be understood by thoseskilled in the art that various modifications may be made tocombinations of the foregoing components and processes, and all suchmodifications are also intended to fall within the scope of the presentinvention.

Embodiment 2 has dealt with hierarchization of frequency division type.However, hierarchization of improved approximation accuracy type may beused instead. In this case, images of various image quality levels canbe decoded by discarding a predetermined number of lower-order bits outof the bit string of the wavelet transform coefficients. That is, thecopyright management unit 40 can adjust the image quality level byinstructing the bit plane decoding unit 256 of the number of bits totransform.

When this method of hierarchization is used, it is possible to make adistinction in image quality level between individual areas of an image.In that case, each of the profiles on the table for managing thecopyright level information shown in FIG. 4 may be described with threeparameters, i.e., application, image quality level, and image area, notthe two parameters or the application and image quality level alone. Theprofiles may also be descried with two parameters of image quality leveland image area. In this case, contents providers can exercise moreflexible copyright management than in the foregoing embodiments.

Different image quality levels can be achieved not only by hierarchizingthe resolution, but by hierarchizing the compression rate or the numberof colors as well. The compression rate can be hierarchized by using thelayering function of JPEG 2000. Color images are expressed by luminancecomponents and color-difference components. Then, the color-differencecomponents may be subjected to the foregoing hierarchization techniques.

Embodiment 5

The technical field of embodiments 5 to 7 is as follows. The inventionrelates to an image display method, an image coding apparatus, an imagedecoding apparatus, and an image display apparatus which can be used toreproduce a moving image acquired through streaming distribution or thelike.

The related art of embodiments 5 to 7 is as follows. In recent years,the prevalence of such infrastructures as DVD media, digitalbroadcasting, and on-demand network communications has producedwidespread use of digital contents. Since digital contents cause nodegradation in quality even after duplication, copyright management isof high importance. Demands for legitimate protection of copyright ondigital contents and other copyrighted materials are expected to grow inthe future.

To protect copyright on images, Japanese Patent Laid-Open PublicationNo. Hei 9-163306 discloses a technique for setting degrees of permissionof information reproduction (hereinafter, referred to as protect levels)stepwise and reproducing contents depending on users of the contents.More specifically, moving images of higher quality are distributed tousers of higher protect levels, and moving images of degraded qualityare distributed to users of lower protect levels.

With the prevalence of the foregoing infrastructures, image contentsproviders are growing in number and the competition is becoming tough.These providers are competing fiercely in expanding sales of their imagecontents, while being aware of the importance of copyright protection.Image contents providers who carry advertisements as their main sourcesof income are also pursuing an increasing number of users who use thecontents provided.

The problems to be solved by embodiments 5 to 7 are as follows. InJapanese Patent Laid-Open Publication No. Hei 9-163306 mentioned above,the users of lower protect levels can only view low quality movingimages, not the original high quality images. With low quality movingimages, it is difficult for the contents to impress the users of lowerprotect levels. Those contents have thus been not quite effective interms of advertisement, nor adequate to promote the purchase of thecontents and the subscription to contents distribution services.

The present invention has been achieved in view of the foregoingcircumstances. It is thus another object of the present invention toprovide an image display method, an image coding apparatus, an imagedecoding apparatus, and an image display apparatus which can performboth protection of copyright and promotion of use of image contents.

The means for solving the problems of embodiments 5 to 7 are as follows.To solve the foregoing problems, an image display method according toone of the aspects of the present invention comprises: displaying amoving image while changing its quality with time in accordance with apredetermined setting value. The “quality” may be degraded. According tothis aspect, image providers can show both high and low quality imagesto users. It is therefore possible to impress users with the highquality image while providing copyright protection on the imagecontents.

The setting value may be one for reducing at least one of resolution,image quality, and a frame rate of the moving image stepwise. At leastone of a luminance and color differences of the moving image may bereduced stepwise. A display area of the moving image may be narroweddown stepwise. The “display area” may be narrowed by reducing the sizeof the image to be displayed itself. “Stepwise” may cover the situationswhere the moving image has three or more quality states. The visiblearea may be reduced without changing the size of the image itself.According to this aspect, the image to be shown to users can be adjustedin quality, and the quality can be lowered for copyright protection.

A predetermined advertisement may be displayed outside the display areaof the moving image. According to this aspect, image providers canobtain advertising opportunities and sources of income.

The setting value may be fixed in response to a predetermined action ofa user. The “predetermined action” may be data transmission forinforming an image provider of legal intention to pay a fee. The“setting value” may be fixed to a value for normal reproduction.According to this aspect, cancellation of copyright management can beassociated with fee acquisition, thereby promoting sales and the like ofimage contents.

Another aspect of the present invention is an image coding apparatus.This apparatus comprises: a coding unit which codes moving image data;and an adding unit which adds a predetermined setting value to the codedmoving image data so that the coded moving image data, when displayed,changes in quality with time. According to this aspect, image providerscan show both high and low quality images to users. It is thereforepossible to impress users with the high quality image while providingcopyright protection on the image contents.

Yet another aspect of the present invention is also an image codingapparatus. This apparatus comprises: a coding unit which codes movingimage data so that the data, when displayed, changes in quality withtime; and a quality setting unit which supplies the coding unit with asetting value for changing the quality. According to this aspect, imageproviders can show both high and low quality images to users. It istherefore possible to impress the users with the high quality imagewhile providing copyright protection on the image contents. Besides, aperiod for degrading the quality can be established to reduce the volumeof the coded image data to be distributed.

Yet another aspect of the present invention is an image decodingapparatus. This apparatus comprises: a decoding unit which decodes codedmoving image data; and a quality setting unit which supplies thedecoding unit with a setting value set so that quality changes withtime. The decoding unit decodes the data while changing its quality inaccordance with the setting value. According to this aspect, highquality images of image contents can be viewed for a certain period evenwhen the contents are under copyright management. This makes it easierfor users to evaluate the image contents.

Yet another aspect of the present invention is an image displayapparatus. This apparatus displays a moving image while changing itsquality with time in accordance with a predetermined setting value.According to this aspect, high quality images of image contents can beviewed for a certain period even when the contents are under copyrightmanagement. This makes it easier for users to evaluate the imagecontents.

Incidentally, any combinations of the foregoing components, and anyexpressions of the present invention converted between methods,apparatuses, systems, computer programs, recording medium, and the likeare also intended to constitute applicable aspects of the presentinvention.

Now, the present invention will be overviewed. Contents creators orcopyrighters configure settings for changing such parameters as imagequality and a frame rate so that moving images are reproduced atdegraded quality for an arbitrary period. Viewers can view the images inoriginal quality by making some kind of contract such as fee payment tocontents providers. The parameters to be changed include resolution,image quality, a frame rate, and color. The display area may be narroweddown. In that case, messages for billing purpose and advertisements maybe displayed outside the area.

Description will first be given of an example for achieving theforegoing, where coded image data generated with no quality adjustmenton the coding side is distributed to the decoding side, and the movingimage is displayed while the quality is adjusted on the decoding side.

FIG. 7 is a block diagram of an image coding apparatus 1100 according toembodiment 5. In terms of hardware, the image coding apparatus 1100 canbe achieved by an arbitrary computer CPU, a memory, and other LSIs. Interms of software, it can be achieved by a program which is loaded on amemory and has coding functions. The functional blocks shown here areachieved by the cooperation of these. It will thus be understood bythose skilled in the art that these functional blocks may be achieved invarious forms including hardware alone, software alone, and combinationsof these.

An orthogonal transform unit 1010 applies a wavelet transform, adiscrete cosine transform, or the like to an input original image. Forexample, in the case of a wavelet transform, the orthogonal transformunit 1010 divides the input original image into subbands, calculates thewavelet transform coefficients of the individual subband images, andgenerates hierarchical wavelet coefficients. Specifically, theorthogonal transform unit 1010 applies a low-pass filter and a high-passfilter to the original image both in x and y directions, therebydividing the original image into four frequency subbands for wavelettransform. These subbands consist of an LL subband, an HL subband and anLH subband, and an HH subband. The LL subband contains low frequencycomponents both in the x and y directions. The HL subband and the LHsubband contain low frequency components in either one of the x and ydirections and high frequency components in the other direction. The HHsubband contains high frequency components both in the x and ydirections. Each subband has as many pixels as ½ those of theunprocessed image both in the vertical and horizontal directions. Asingle application of the filtering thus produces subband images havinga resolution, i.e., image size of ¼.

Among the subbands obtained thus, the orthogonal transform unit 1010applies filtering to the LL subband again, thereby dividing the sameinto LL, HL, LH, and HH four subbands further for wavelet transform. Theorthogonal transform unit 1010 performs this filtering a predeterminednumber of times to transform the original image into hierarchicalsubband images, and outputs the wavelet transform coefficients of therespective subbands. A quantization unit 1012 quantizes predeterminedcoefficients, such as the wavelet transform coefficients output from theorthogonal transform unit 1010, at predetermined quantization widths.

An entropy coding unit 1014 entropy-codes the values quantized by thequantization unit 1012. For example, it scans and codes the quantizedvalues from the upper bit planes in succession. As can be seen, thetarget to be coded by the entropy coding unit 1014 is the originalimage. The entropy coding unit 1014 thus functions as an image codingunit.

A quality information adding unit 1018 adds information for controllingthe quality with which the stream coded by this image coding apparatus1100 is decoded and displayed (hereinafter, referred to as qualityinformation), to the coded stream. This quality information defines atleast any one of various parameters including the resolution, imagequality, frame rate, display area, color differences, and brightness fordecoding, in association with a lapse of time schedule during decoding.For example, in the case of the resolution, the resolution can be set todecrease with time during decoding and display. For instance, an imageis decoded at full resolution for five minutes since the start of thedecoding, decoded without high frequency components for the next fiveminutes, and decoded without medium and high frequency components in thenext five minutes. Eventually, direct-current components alone may bedecoded and displayed. This decoding method is suited to coded streamdecoding having a SVC (Scalable Video Codec) function such as theforegoing wavelet transform.

In the case of controlling the image quality, the numbers of bits of themulti-bit coefficients, such as wavelet transform coefficients anddiscrete cosine transform coefficients, to be decoded may be decreasedwith time. Image quality can be degraded by quitting decoding lower bitsgradually.

The display area can be controlled by establishing a region of interest(ROI) in the image. Information for specifying the region of interest(hereinafter, referred to as ROI information) includes position, shape,size, and image quality. This ROI information may also be set so thatthe size decreases with time. The contents of the quality informationwill be detailed later along with other parameters.

The quality information adding unit 1018 can write to coded streamheaders. There are various levels of headers, and any of them may beused for that purpose. For example, the quality information may bewritten to a stream header, a sequence header, a GOP (Group of Picture)header, a frame header, a picture header, and so on.

The quality information adding unit 1018 can generate information fordisabling quality management on the decoding side depending on suchfactors as a fee payment by a user on the decoding side. Thisinformation is added to the coded data of the image in a coded streamgeneration unit 1016 to be described later, or transmitted to the user'simage decoding apparatus to be described later, thereby disabling thequality management on the moving image in the image decoding apparatus.

The coded stream generation unit 1016 generates a coded stream based onthe coded data of the image input from the entropy coding unit 1014, thecoded data on the quality information input from the quality informationadding unit 1018, etc. The coded stream generation unit 1016 thenoutputs the coded stream generated to a network or a recording medium.

As has been described, according to the present embodiment, it ispossible to achieve an image coding apparatus which can perform bothprotection of copyright and promotion of use of moving images. Imageproviders can protect copyright by degrading the quality of the movingimages with time. Besides, it is possible to show high quality imagesfor a certain period and impress users, thereby promoting the sales ofthe contents and the subscription to contents distribution services. Forexample, the beginning of a program is distributed for free and in highquality in order to attract viewers, and then the quality is graduallylowered in the middle of the program. This can fuel the contents-buyingmotivations of viewers who want to view the whole program.

Moreover, image providers can create contents easily with a reduction inmemory area since moving image data need not be provided for trialpurpose and for reproduction purpose separately. Furthermore, in thecase of narrowing down the display area, it is possible to use theremaining space for advertisements of their own or other companies, andthereby earn advertising income.

Embodiment 6

FIG. 8 is a block diagram of an image decoding apparatus 1200 accordingto embodiment 6. The image decoding apparatus 1200 decodes a codedstream having additional quality information, such as the coded streamcoded by the foregoing image coding apparatus 1100.

A stream acquisition unit 1020 acquires a coded stream from exterior,and stores it into a stream memory 1022 temporarily. The coded streamacquired may be one that is transmitted through streaming distribution.It may also be one that is recorded on a recording medium such as aDVD-ROM.

When a stream level for decoding is set by a quality setting unit 1036to be described later, the stream acquisition unit 1020 discards some ofthe coded stream acquired and passes the remaining to a variable lengthdecoding unit 1024 in accordance with that stream level. For example,high frequency components may be discarded for lowering its resolution.

The stream memory 1022 stores the coded stream input from the streamacquisition unit 1020 temporarily. The variable length decoding unit1024 decodes the coded stream passed from the stream acquisition unit1020 with respect to each bit plane, decodes quantized pixel data andvarious parameters, and outputs the resultant to an inverse quantizationunit 1026. For a stream coded by inter-frame prediction coding, motionvectors are also decoded. For an entropy-coded stream, entropy decodingis performed.

The inverse quantization unit 1026 inversely quantizes the input pixeldata, and outputs the resultant to an inverse orthogonal transform unit1028. If a quantization scale is set by a quantization scale settingunit 1038 to be described later, the inverse quantization is performedbased on the quantization scale. The inverse orthogonal transform unit1028 restores pixel values that are transformed by a discrete cosinetransform, a wavelet transform, or the like. For a stream coded byinter-frame prediction coding, difference values of corresponding pixelsbetween frames are restored. The inverse orthogonal transform unit 1028outputs these pixel values and the like to a pixel value calculationunit 1030. The pixel value calculation unit 1030 writes the restoredpixel values into an image memory 1042.

The image memory 1042 retains the pixel values restored by the pixelvalue calculation unit 1030 temporarily. Incidentally, the image memory1042 may use the same memory as the stream memory 1022 if the addressspaces are different.

A reference data specification unit 1040 is necessary for a stream codedby inter-frame prediction coding such as MPEG. For reference values, thereference data specification unit 1040 specifies pixel values of thereference image retained in the image memory 1042 that are designated bythe motion vectors passed from the variable length decoding unit 1024.In that case, the pixel value calculation unit 1030 adds the referencevalues specified by the reference data specification unit 1040 and thecorresponding difference values acquired from the inverse orthogonaltransform unit 1028, thereby restoring pixel values in the image memory1042. The pixel value calculation unit 1030 also corrects luminancevalues or color differences when it is instructed of luminance orcolor-difference correction from the quality setting unit 1036.

A frame reconstruction unit 1032 assembles the pixel values restored bythe pixel value calculation unit 1030 frame by frame to reconstructmoving image data. When a frame rate is set by the quality setting unit1036, the frame reconstruction unit 1032 reconstructs the moving imagedata at that frame rate. When a display size is set by the qualitysetting unit 1036, pixels are skipped or interpolated. A display unit1050 displays the moving image data reconstructed by the framereconstruction unit 1032. Incidentally, the display unit 1050 may be anexternal device. In that case, the image decoding apparatus 1200 decodesthe coded stream and outputs the decoded data to the not-shown displaywhich is in a cable or wireless connection.

The quality management unit 1034 acquires quality information that isadded to the coded stream input from exterior. When information fordisabling the foregoing quality management is delivered separately, thequality management unit 1034 also acquires it. Based on the qualityinformation acquired, the quality management unit 1034 manages theelapsed time of reproduction of the moving image, the billing status,and the like, and instructs the quality setting unit 1036 of the qualityspecified by this quality information.

Based on the quality instructed by the quality management unit 1034, thequality setting unit 1036 sets quality-adjusting parameters to theindividual units. Initially, in the case of changing the resolution, thequality setting unit 1036 sets a stream level to the stream acquisitionunit 1020. FIG. 9 is a diagram showing an example where the resolutionis lowered with time. The diagram shows the lapse of time from left toright. The left screen 1060 shows a screen that is decoded by using allthe frequency components included in a coded stream. The center screen1062 shows one that is decoded while discarding high frequencycomponents of the coded stream. The right screen 1064 shows one that isdecoded by using only direct-current components and low frequencycomponents of the coded stream. As can be seen, the nonuse of the highfrequency components lowers the resolution, and the use of the highfrequency components raises the resolution. A coded stream having an SVCfunction can be changed in resolution easily by specifying, as a streamlevel, which frequency components to use.

In the case of changing image quality, the quality setting unit 1036sets a quantization scale to the quantization scale setting unit 1038.FIG. 10 is a diagram showing an example where the image quality islowered with time. The image quality decreases from the left screen 1070to the right screen 1074. The quality setting unit 1036 sets aquantization scale to the quantization scale setting unit 1038 so thatthe quantization scale decreases gradually. When an ROI function isimplemented as in JPEG 2000, the entire screen may be set as the ROIarea and the ROI information may be modified gradually.

In the case of changing the display area, the quality setting unit 1036sets ROI information to the quantization scale setting unit 1038. FIG.11 is a diagram showing an example where the display area is narroweddown with time. The display area decreases from the left screen 1080 tothe right screen 1084. The quality setting unit 1036 narrows down theROI area toward the center. The areas other than the ROI area may beblackened, or used for CM areas as shown in FIG. 11. Advertisements suchas logotypes and messages to be displayed in the CM areas may be codedalong with the image in advance, or may be superimposed by the pixelvalue calculation unit 1030 of this image decoding apparatus 1200.

Incidentally, the processing of FIG. 11 and the processing of FIG. 9 or10 can be combined to maintain the resolution or image quality of thearea that is narrowed toward the center, while decreasing the resolutionor image quality of the other areas.

In the case of changing the size of the image to display, the qualitysetting unit 1036 sets a display size to the frame reconstruction unit1032. FIG. 12 is a diagram showing an example where the image is scaleddown with time. The image is scaled down from the left screen 1090 tothe right screen 1094. The quality setting unit 1036 need not make anyoperation if the display size specified by the quality management unit1034 is the same as the size of the moving image restored. If thedisplay size is smaller than the size of the moving image restored, thequality setting unit 1036 instructs the frame reconstruction unit 1032to skip pixels at a predetermined ratio. If the display size specifiedis greater than the size of the moving image restored, additional pixelsare calculated from adjoining pixel values. If this scaling processingcan cause a severe burden on the system, a dedicated scaling hardwarecircuit may be provided for that processing.

In the case of changing the frame rate, the quality setting unit 1036sets a frame rate to the frame reconstruction unit 1032. FIG. 13 is adiagram showing an example where the frame rate is reduced with time.Note that the frame rate refers to the frequency of update of frames. InFIG. 13, the frame rate is reduced with a lapse of time. The frame ratefor an identical period is reduced from eight frames to four frames, andto two frames. Incidentally, a frame shows a still image unless updated.The quality setting unit 1036 can thus specify a still image.

In the case of changing the brightness of the display image, the qualitysetting unit 1036 sets a luminance correction value to the pixel valuecalculation unit 1030. The pixel value calculation unit 1030 adds orsubtracts the correction value to/from the original luminance values.The addition makes the image brighter, and the subtraction dimmer.

In the case of changing the color of the display image, the qualitysetting unit 1036 sets color-difference correction values to the pixelvalue calculation unit 1030. The pixel value calculation unit 1030corrects color differences based on the correction values. Correctingthe original color-difference values toward zero can bring the correctedimage nearer to a monochrome. Color-difference values of zero produce amonochrome image.

The quality setting unit 1036 may change two or more of the foregoingparameters in combination. When information for disabling qualitymanagement is received, the quality setting unit 1036 resets theforegoing parameters to their normal values. Incidentally, the qualityat the beginning of reproduction of a moving image can also be setarbitrarily. The quality may also be set to increase with time dependingon applications, such as when it is desired to display the end-titlecredit of a movie, the last part of an image, or the like clearly.

As has been described, according to the present embodiment, it ispossible to achieve an image decoding apparatus which can perform bothprotection of copyright and promotion of use of moving images. Imageproviders can protect copyright by degrading the quality of the movingimages with time. Besides, it is possible to show high quality imagesfor a certain period and impress users, thereby promoting the sales ofthe contents and the subscription to contents distribution services.Moreover, when the display area is narrowed down, it is possible to usethe remaining space for advertisements of their own or other companies,and thereby earn advertising income.

Users can view high quality images for a certain period to obtaininformation for making an appropriate decision whether or not topurchase that contents. Users can sign a contract with the contentsproviders to reproduce the moving image with normal quality. Displayingadvertisements outside the display area increases the possibility forusers to access the contents at small or no fee. This mechanism is thushighly advantageous even for users.

Now, description will be given of an example where coded image data ofadjusted quality is generated on the coding side before distributed tothe decoding side.

Embodiment 7

FIG. 14 is a block diagram of an image coding apparatus 1300 accordingto embodiment 7. Incidentally, the image coding apparatus 1300 iscapable of inter-frame coding. This is common processing, and thus thecorresponding parts are omitted from FIG. 14.

If a quality setting unit 1318 to be described later makes a change toluminance values and/or color-difference values, a pixel valuecorrection unit 1302 corrects those values in accordance with settingvalues set by the quality setting unit 1318. An image memory 1304 isused for the correction. If no change will be made to the luminancevalues and the like, neither of the pixel value correction unit 1302 andthe image memory 1304 need to be provided.

An orthogonal transform unit 1306 performs wavelet transforms, discretecosine transforms, or the like. If resolution is set by the qualitysetting unit 1318 to be described later, the number of wavelettransforms is adjusted accordingly, for example. A quantization unit1308 quantizes predetermined coefficients, such as the wavelet transformcoefficients output from the orthogonal transform unit 1306, atpredetermined quantization widths. If a quantization scale is set by aquantization scale setting unit 1320 to be described later, thequantization is performed based on the quantization scale. A variablelength coding unit 1310 codes a bit string passed from the quantizationunit 1308 bit plane by bit plane. The variable length coding unit 1310also codes various parameters necessary for decoding. Incidentally, thebit string may be entropy-coded.

A stream generation unit 1312 assembles the coded image data input fromthe variable length coding unit 1310, thereby generating a coded stream.Then, the coded stream generated is output to a network or a recordingmedium. If a frame rate or display size is set by the quality settingunit 1318 to be described later, the stream generation unit 1312 changesthe frame rate or display size accordingly. Here, the stream generationunit 1312 can use a stream memory 1314. Decoding apparatuses thatreceive the foregoing coded steam can decode it normally.

A quality management unit 1316 manages various types of quality set bythe image provider so that the moving image changes in quality withtime, and sets the same to the quality setting unit 1318. When qualitymanagement is cancelled due to such reasons as a purchase of thecontents by a viewer, the quality management unit 1316 also informs itto the quality setting unit 1318.

Based on the quality set by the quality management unit 1316, thequality setting unit 1318 sets quality-adjusting parameters to theindividual units. In the case of changing the resolution, the qualitysetting unit 1318 sets the number of wavelet transforms to theorthogonal transform unit 1306. In the case of changing the imagequality, the quality setting unit 1318 sets a quantization scale to thequantization scale setting unit 1320.

In the case of changing the display area, the quality setting unit 1318sets ROI information to the quantization scale setting unit 1320. In thecase of changing the size of the image to display, the quality settingunit 1318 sets a display size to the stream generation unit 1312. In thecase of changing the frame rate, the quality setting unit 1318 also setsa frame rate to the stream generation unit 1312.

In the case of changing the brightness of the display image, the qualitysetting unit 1318 sets a luminance correction value to the pixel valuecorrection unit 1302. In the case of changing the color of the displayimage, the quality setting unit 1318 sets color-difference correctionvalues to the pixel value correction unit 1302. Like the quality settingunit 1036 described in embodiment 6, the quality setting unit 1318 canalso set parameters to the individual units. Each of the units can alsochange the quality of the moving image similarly.

As has been described, according to the present embodiment, it ispossible to achieve an image coding apparatus which can perform bothprotection of copyright and promotion of use of moving images. When themoving image data of degraded quality is generated on the coding side,it is possible to reduce the volume of the coded data. This can reducethe amount of traffic on the network, and save the recording capacitiesof the hard disks or the like on the receiving user terminals. Moreover,it is possible to generate highly-versatile coded data which can bedecoded by existing decoding apparatuses.

Up to this point, the present invention has been described inconjunction with several embodiments thereof. The foregoing embodimentshave been given solely by way of example. It will thus be understood bythose skilled in the art that various modifications may be made tocombinations of the foregoing components and processes, and all suchmodifications are also intended to fall within the scope of the presentinvention.

In the foregoing embodiments, a server independent from the image codingapparatuses 1100 and 1300 may manage the billing statuses of users. Userpayments, when made, may be informed to the image coding apparatuses1100 and 1300 to cancel the quality management. In this case, existingbilling systems can be used.

The foregoing embodiments 5 and 6 have dealt with the cases where thequality information is added to the coded stream and transmitted to thedecoding side. Nevertheless, the quality information may be downloadedto the decoding side separately. Spontaneous user downloading can beexpected if reproduction is prohibited without this information. Thiseliminates the need for the coding side to add the quality informationto the coded stream, whereby the configuration of the coding apparatuscan be simplified.

Embodiment 8

The technical field of embodiment 8 is as follows. The invention relatesto an image processing apparatus, an image display apparatus, and animage display method for decoding a coded image data stream having ahierarchical structure.

The related art of embodiment 8 is as follows. In recent years, theprevalence of such infrastructures as DVD media, digital broadcasting,and on-demand network communications has produced widespread use ofdigital contents. Since digital contents cause no degradation in qualityeven after duplication, copyright management is of high importance.Demands for legitimate protection of copyright on digital contents andother copyrighted materials are expected to grow in the future. JapanesePatent Laid-Open Publication No. 2001-86444 discloses a technique forprohibiting duplication by using copy control information included in astream.

The problems to be solved by embodiment 8 are as follows. Techniqueslike this are no more than a uniform approach of either permitting orprohibiting, and it is difficult to satisfy various levels of demandsboth from image providers and from users. For example, image providersmight lose business opportunities if they simply prohibit duplication.Besides, users have various orientations ranging from high-end toprice-sensitive.

The present invention has been achieved in view of the foregoingcircumstances. It is thus an object of the present invention to providean image processing apparatus, an image display apparatus, and an imagedisplay method capable of flexible provision of images which isbeneficial to both image providers and users who use the images.

The means for solving the problems of embodiment 8 are as follows. Tosolve the foregoing problems, an image processing apparatus according toone of the aspects of the present invention comprises: a decoding unitwhich acquires coded image data multiplexed so as to be decodable in aplurality of image qualities, and decodes the coded image data so as toproduce at least two or more images having different image qualities;and a memory unit which stores a plurality of pieces of image datadecoded in order to cope with predetermined processing. The “imagequality” may include resolution. The “predetermined processing” mayinclude displaying a plurality of images having respective differentimage qualities. In that case, the images may be displayed in amulti-window fashion.

According to this aspect, it is possible to make flexible provision ofimages which is beneficial to both image providers and users who use theimages. In addition, a plurality of images having different imagequalities can be processed in real time.

Another aspect of the present invention is also an image processingapparatus. This apparatus comprises: a decoding unit which acquirescoded image data multiplexed so as to be decodable in a plurality ofimage qualities, and decodes the coded image data in a plurality ofimage qualities decodable; and a memory unit which stores a plurality ofpieces of image data decoded in order to cope with predeterminedprocessing.

According to this aspect, it is possible to make flexible provision ofimages which is beneficial to both image providers and users who use theimages. In addition, a plurality of images having different imagequalities can be processed in real time.

The decoding unit may decode the coded image data in all the imagequalities decodable. According to this aspect, it is possible to createa situation where images can be provided from the coded image data themost flexibly.

The image processing apparatus may further comprise a copyrightmanagement unit which exercises copyright management on the plurality ofpieces of image data stored in the memory unit in accordance with setcopyright management information. The “copyright management information”may define a permitted image quality level when using the coded imagedata, or define a combination of an application and a permitted imagequality level of the coded image data. According to this aspect,flexible copyright management becomes possible. For example, it ispossible to establish image qualities to be subjected to the copyrightmanagement and ones not to.

The copyright management unit may add a noise component to at least oneof the plurality of pieces of image data stored in the memory unit.According to this aspect, flexible copyright management becomespossible, instead of such copyright management as prohibiting useuniformly.

The image processing apparatus may further comprise a transmission unitwhich transmits image data selected from among the plurality of piecesof image data stored in the memory unit to a predetermined terminal. Thetransmission unit may transmit pieces of image data having differentimage qualities to a plurality of terminals, respectively, atsubstantially the same time. The transmission unit may transmit imagedata selected by a user operation, or transmit image data selected basedon specification information on a predetermined terminal. Thisspecification information may include resolution of a display unit ofthat terminal. According to this aspect, flexible image transmissionbecomes possible.

Another aspect of the present invention is an image display apparatus.This apparatus comprises: an image processing apparatus according to anyone of the foregoing aspects; and a display unit which displays aplurality of images having respective different image qualities.

According to this aspect, it is possible to make flexible display ofimages which is beneficial to both image providers and users who use theimages. In addition, a plurality of images having different imagequalities can be displayed in real time.

The display unit may display the plurality of images having differentimage qualities while copyrights on the respective images are managedindependently in accordance with set copyright management information.According to this aspect, flexible copyright management becomespossible.

The display unit may display the plurality of images having differentimage qualities in a multi-window fashion so as to have a user select animage of any one of the image qualities. According to this aspect, it ispossible to improve the user operability.

Yet another aspect of the present invention is an image display method.This method comprises displaying a plurality of images having respectivedifferent image qualities decoded from coded image data multiplexed soas to be decodable in a plurality of image qualities.

According to this aspect, it is possible to make flexible display ofimages which is beneficial to both image providers and users who use theimages. In addition, a plurality of images having different imagequalities can be displayed in real time.

Incidentally, any combinations of the foregoing components, and thecomponents and expressions of the present invention mutually replacedwith methods, apparatuses, systems, recording medium, programs, and thelike are also intended to constitute applicable aspects of the presentinvention.

By way of example, a method for coding a moving image by usingMotion-JPEG 2000 scheme will be described briefly with reference to FIG.15. A not-shown image coding apparatus generates a coded image datastream of a moving image by coding individual frames of the moving imagecontinuously frame by frame. At the beginning of the coding process, anoriginal image OI 102 corresponding to a single frame of the movingimage is read into a frame buffer. The original image OI read into theframe buffer is hierarchized by a wavelet transformer.

The JPEG-2000 wavelet transformer uses a Daubechies filter. This filterfunctions as a high-pass filter and a low-pass filter both in x and ydirections of an image simultaneously, thereby dividing the single imageinto four frequency subbands. These subbands consist of an LL subband,an HL subband and an LH subband, and an HH subband. The LL subbandcontains low frequency components both in the x and y directions. The HLsubband and the LH subband contain low frequency components in eitherone of the x and y directions and high frequency components in the otherdirection. The HH subband contains high frequency components both in thex and y directions. This filter also has the function of reducing thenumber of pixels by half both in the x and y directions. That is, eachsubband has as many pixels as a half those of the unprocessed image bothin the vertical and horizontal directions. A single application of thefiltering thus produces subband images having a resolution, i.e., imagesize of ¼. As employed in this specification, an image obtained byapplying a single wavelet transform to the original image OI will bereferred to as a first-level image WI1. Hereinafter, an nth-level imagewill be referred to as WIn in accordance with the number of wavelettransforms applied thereto.

As schematically shown in FIG. 15, the first-level image WI1 104 hasfour subbands LL1, HL1, LH1, and HH1. A wavelet transform is applied tothe first-level image WI1 104, whereby a second-level image WI2 106 isgenerated. The second and subsequent wavelet transforms will be appliedto only the LL subband components of the images in the respectiveprevious levels. Consequently, in the second-level image WI2 106, theLL1 subband of the first-level image WI1 is decomposed into foursubbands LL2, HL2, LH2, and HH2. The wavelet transformer performs thisfiltering a predetermined number of times, and outputs the wavelettransform coefficients of the respective subbands. The image codingapparatus then performs quantization and other processing, and finallyoutputs coded image data CI.

For ease of explanation, the image coding apparatus in this exampleshall apply three wavelet transforms to the original image OI. Suppose,for example, that the original image OI 102 has 1440×960 pixels. Then,the LL1 subband of the first-level image WI1 104 has a size of 720×480,the LL2 subband of the second-level image WI2 106 a size of 360×240, andthe LL3 subband of the third-level image WI3 108 a size of 180×120.

As far as the hierarchical images are concerned, it should be noted thatthe low frequency components of the original image OI gather around theupper left in FIG. 15. In the case of FIG. 15, the LL3 subband whichfalls on the upper left corner of the third-level image WI3 is thelowest in frequency. Conversely, the most basic properties of theoriginal image OI can be reproduced as long as this LL3 subband isaccessible.

Aside from Motion-JPEG 2000, the coded data stream may be of, e.g., SVC(Scalable Video Codec) which provides a single stream including both ahigh quality HD stream and a low quality SD stream in combination.Motion-JPEG is also applicable. Among JPEG schemes are ones in whichframes are transmitted in ascending order of the terms of their DCT(Discrete Cosine Transform) coefficients obtained by discrete cosinetransform. Here, an image quality can be selected depending on up towhat order of terms the coefficients are used for decoding. According tothese specifications, spatial resolution can be coded hierarchically.

The foregoing example has been dealt with the case where thehierarchization is achieved by the coding of frequency division type.Nevertheless, coding of improved approximation accuracy type may be usedfor hierarchization. More specifically, the number of higher-order bitsof DCT coefficients or wavelet coefficients to be decoded can beadjusted to achieve decoding in different image qualities. In MPEG-2,not only the foregoing spatial resolution but also time resolution canbe coded hierarchically by adjusting the number of frames.

FIG. 16 shows an example of the data structure of the coded image datastream. The header in FIG. 16 represents a stream header. On the codingside, copyright level information to be described later can be added toa free space in this header area. Each of the frames also has anot-shown frame header which describes information and the likenecessary for decoding. Incidentally, the copyright level informationmay be added to frame headers. The information may be added to any ofthe frames in that group of frames, such as every few frames.

When the hierarchical coded image data CI is generated by frequencydivision as described above, each frame describes data on a lowfrequency component, a medium frequency component, and a high frequencycomponent in succession from the top of the data area. Incidentally,data on a direct-current component appears before that of each lowfrequency component when the scheme in use also divides direct-currentcomponents. It is possible to reproduce images even from the firstportions of the respective frames alone, whereas the images are low inquality. The image improves in quality as the last portions are involvedin the reproduction. As described above, the hierarchical coded imagedata stream has both image data areas necessary for reproducing highquality images, and overlapping image data areas necessary forreproducing low quality images. A plurality of images having differentimage qualities can thus be reproduced from the single coded image datastream.

FIG. 17 shows an example of the copyright level information. Thecopyright level information is intended to manage copyright levels to bepermitted to users who use the coded image data CI. The use of the codedimage data CI is not managed in an either-or fashion, i.e., between fullpermission and prohibition. Instead, independent management is exerciseddepending on user patterns including applications, image qualities, andcombinations of these. In FIG. 17, the copyright level information to beadded to the coded image data CI is expressed in a table. Theapplications of the coded image data CI are shown on the horizontalaxis, and the permitted image quality levels corresponding to therespective applications are shown on the vertical axis. Display, one ofthe applications, is permitted without limitation if in the low imagequality. View-blocking noise such as scrambling is added in the mediumand high image qualities. Display is thus substantially limited in themedium and high image qualities. Incidentally, this noise is no longersuperimposed if a predetermined condition such as a fee payment issatisfied by the user. Initial display represents a function fordisplaying an image that should later be hardly visible due tosuperimposed noise, onto a noise-free normal screen for a certain periodsince the beginning of display. In FIG. 17, one minutes of initialdisplay is provided in the high image quality, and 30 seconds in themedium image quality. Noise will be superimposed on the imagethereafter.

Transmission represents a function for transmitting the coded image dataCI from the decoding device to another device. In FIG. 17, transmissionis permitted without limitation in all the image qualities. Duplicationis prohibited in the high image quality, and is permitted withoutlimitation in the low and medium image qualities. Edit is prohibited inthe medium and high image qualities, and is permitted without limitationin the low image quality. This editing may include such operations asremoving commercial breaks from coded image data that is received in adigital television broadcast. In that case, the coded image data can beduplicated in the medium image quality, but not when commercial breaksare removed.

Image providers can manage such profile tables user by user, and updatethe individual items in the profiles if necessary. When the coded imagedata CI is provided to paid users, different profiles may be assigneddepending on fees paid by the users, price plans selected, etc. Theavailable applications and the permitted image levels for thoseapplications also vary depending on the profiles. The profiles may alsobe created region by region or time by time, instead of user by user.For example, time-based profiles can be created to impose time limits onthe availability of contents for users. If user apparatuses areconfigured to switch profiles after a certain period in time, timelimits can be set so as to lower the permitted image quality levels orrestrict some uses, instead of making all the contents unavailable afterthat period.

Now, description will be given of an image processing apparatus 2200which decodes a coded image data stream hierarchized as described above.FIG. 18 shows the configuration of the image processing apparatus 2200according to embodiment 8 of the present invention. In terms ofhardware, the image processing apparatus 2200 can be achieved by anarbitrary computer CPU, a memory, and other LSIs. In terms of software,it can be achieved by a program which is loaded on a memory and hasdecoding functions. The functional blocks shown here are achieved by thecooperation of these. It will thus be understood by those skilled in theart that these functional blocks may be achieved in various formsincluding hardware alone, software alone, and combinations of these.

The image processing apparatus 2200 comprises a decoding block 2250. Thedecoding block 2250 includes a stream analysis unit 2252, an arithmeticdecoding unit 2254, a bit plane decoding unit 2256, an inversequantization unit 2258, an inverse wavelet transform unit 2260, and aframe buffer 2262.

The stream analysis unit 2252 acquires a coded image data stream CI fromexterior, and analyzes the data stream. The coded image data stream CImay be downloaded over a network, acquired via airwaves, or acquiredfrom a recording medium containing the same. If copyright levelinformation is added to a header or the like of the coded image data CI,the stream analysis unit 2252 detects and passes it to a copyrightmanagement unit 2030.

The arithmetic decoding unit 2254 performs a stream analysis, andapplies arithmetic decoding to the resulting data string to be decoded.The bit plane decoding unit 2256 decodes the data resulting from thearithmetic decoding into bit planes with respect to respective colorcomponents. The inverse quantization unit 2258 inversely quantizes thequantized data decoded. The inverse wavelet transform unit 2260 appliesinverse wavelet transforms to the nth-level image WIn resulting from theinverse quantization, by using the frame buffer 2262. Each time aninverse wavelet transform is applied to the coded image data CI, animage of the higher level is obtained. Decoding up to the topmost levelproduces decoded image data DI.

The decoding block 2250 outputs a plurality of levels of image datadeveloped in the frame buffer 2262 to a memory unit 2034. For example,the image data developed in all the levels from the first to the topmostlevel may be output. In this case, all the levels of image data can beoutput by outputting the image data in the frame buffer 2262 to thememory unit 2034 each time an inverse wavelet transform is performed.Otherwise, image data in two or more levels out of the first to thetopmost levels may be output to the memory unit 2034 in accordance withpredetermined settings. For example, the first to the topmost levels maybe output alternately.

The copyright management unit 2030 manages copyrights on the coded imagedata CI to be used in this image processing apparatus 2200. Thecopyright management unit 2030 manages the copyright level informationpassed from the stream analysis unit 2252 or copyright level informationCLI acquired from an image provider's server over the Internet or othernetwork. Such copyright level information may include information forpermitting uses that are so far limited, depending on such factors as afee payment. For example, information for permitting display in the highimage quality may be included if it is so far limited. When the useruses this coded image data CI, the copyright management unit 2030identifies the permitted image quality corresponding to the application,and manages a plurality of pieces of image data having different imagequalities stored in the memory unit 2034.

The copyright management unit 2030 may include a noise superimpose unit2032. The noise superimpose unit 2032 superimposes a predetermined noisecomponent on the decoded image data so as to generate a view-blockingdisplay such as scrambling. The noise component may be superimposedwhile inverse wavelet transforms are performed in the frame buffer 2262of the decoding block 2250. It may also be superimposed when the imagedata stored in the memory unit 2034 is further output to thetransmission unit 2036 or the display unit 2038. Incidentally, the noisesuperimpose unit 2032 need not be provided if the copyright managementis only exercised by simply prohibiting image display in certain imagequalities. Moreover, when this image processing apparatus 2200 will notmake any copyright management, the copyright management unit 2030 neednot be provided.

The memory unit 2034 stores the image data output from the decodingblock 2250. In the present embodiment, the memory unit 2034 can store aplurality of pieces of image data having different image qualities, andprovide images having the same contents but different image qualities tothe transmission unit 2036 and/or the display unit 2038 simultaneously.

FIG. 19 shows the data structure of the memory unit 2034 according tothe present embodiment 8. The memory unit 2034 stores a plurality ofpieces of image data having different image qualities which are decodedfrom a single coded image data CI. In FIG. 19, high resolution imagedata 2322, medium resolution image data 2324, and low resolution imagedata 2326 are stored in respective independent forms. In the case ofmoving images, the pieces of image data in the respective areas areupdated in succession, whereas the pieces of image data stored aresubstantially of the same scene.

The transmission unit 2036 transmits the image data stored in the memoryunit 2034 to such terminals as a TV set, PC, a cellular phone, and a PDA(Personal Digital Assistance) via wired or wireless communications. Thedisplay unit 2038 displays the image data stored in the memory unit2034. Incidentally, this image processing apparatus 2200 has only tohave at least either one of the transmission unit 2036 and the displayunit 2038.

A selection unit 2040 selects which image data to output from among theplurality of pieces of image data stored in the memory unit 2034 whenoutputting image data from the memory unit 2034 to the transmission unit2036 or the display unit 2038. The selection unit 2040 may select datain response to a user instruction from a not-shown operation unit, orbased on predetermined specification information. The specificationinformation may include the resolution of the display unit 2038, theresolution of the display on a destination of transmission, thecomputing speed of the decoding block 2250, and the memory capacity ofthe same. For example, the selection unit 2040 may acquire theresolution of the display of a destination terminal in advance, so thatimage data having a highest resolution within that value can be selectedautomatically.

Next, description will be given of an example of operation of the imageprocessing apparatus 2200 according to the present embodiment 8. Sinceit holds a plurality of pieces of image data having different imagequalities in the memory unit 2034, the image processing apparatus 2200can realize various applications as described below.

First, this image processing apparatus 2200 can display images havingthe same contents but different image qualities at substantially thesame time. FIG. 20 shows a screen on which images having the samecontents but different image qualities are displayed in a multi-windowfashion. In FIG. 20, three images, i.e., a high resolution image 2342, amedium resolution image 2344, and a low resolution image 2346 generatedfrom the high resolution image data 2322, the medium resolution imagedata 2324, and the low resolution image data 2326 are displayed tooverlap on a single screen. The user can select which one to displayfrom among the plurality of images having different image qualitiesdisplayed on the display unit 2038. The selection unit 2040 exercisescontrol so that the image data for displaying the user-selected imagealone is output from the memory unit 2034 to the display unit 2038.

A not-shown resizing unit may scale up or down the selected image dataaccording to a user instruction. Note that image data scaled up canmerely cause block noise, not improve the image quality, if itoriginally has low resolution. The image selection screen as shown inFIG. 20 may appear any time in response to user operations.

This configuration allows such processing that a user who is viewing themedium resolution image switches to the high resolution image so as towatch a more dynamic image. This configuration also allows suchapplications as billing different fees depending on the display imagequalities. Fee rates per unit time of display may be determineddepending on the image quality. For example, low quality display can beset at a low rate or for free, and high quality display at a relativelyhigh rate. This allows such viewing mode that a user can select lowquality images as long as simply following the story, and select highquality images when at the climax. To achieve this, a time monitoringunit may be provided. The not-shown time monitoring unit monitors theselection unit 2040, and records in which image qualities and for howmany hours the user views the images as viewing time information. Then,this viewing time information is transmitted to the image provider'sserver or the like after the end of the view, so that the image providercan calculate from that information how much fees to charge to the user.

When this image processing apparatus 2200 displays images having thesame contents but different image qualities at substantially the sametime, view-blocking noise may be superimposed on one or more of theimages. FIGS. 21 shows a screen on which images having the same contentsbut different image qualities, including noise-superimposed one, aredisplayed in a multi-window fashion. In FIG. 21, the high resolutionimage 2342 generated from the high resolution image data 2322 isscrambled. As described above, in accordance with the copyright levelinformation acquired, the copyright management unit 2030 can superimposenoise on images having image qualities that are subject to copyrightmanagement by the image provider. Moreover, such a control asdescrambling the screen for a limited period of time is also possible.For example, it is possible to exercise such a control that an imagehaving the image quality to be managed is displayed normally for severaltens of seconds since the beginning of display. Then, the image may beswitched to a scrambled image. If the user wants to remove thesuperimposed noise from the image and view the image, he/she can selectthe image and make a predetermined procedure such as a fee payment toremove the noise.

This image processing apparatus 2200 is also capable of transmittingimages having the same contents to a plurality of terminals atsubstantially the same time. In this mode of use, the image processingapparatus 2200 can be operated as a home server. Here, images havingdifferent image qualities can be transmitted to respective terminals.Noise-superimposed images can also be transmitted to one or more of theplurality of terminals. Which quality images to transmit to whichterminals can be specified by user operations. The selection unit 2040may also transmit images of optimum image qualities automatically basedon specification information on the destination terminals. For example,low resolution image data may be automatically transmitted to terminalshaving small display areas such as a cellular phone. Users can easilyinstruct the image processing apparatus 2200 to transmit image data totheir cellular phones so that they can view the images even when leavinghome. For example, it is possible to continue watching TV contents evenafter leaving home by using a cellular phone.

As has been described, according to the present invention, it ispossible to realize flexible provision of images which is beneficial toboth image providers and users who use the images. More specifically,since the memory unit stores image data on the same contents but indifferent image qualities, those pieces of image data can be used inreal time to achieve various types of processing. For example, aplurality of images having different image qualities may be displayed ina multi-window fashion. Images having different image qualities may betransmitted to respective different terminals at substantially the sametime. Image providers can utilize these functions to effect varioussales techniques and sales promotion techniques.

Moreover, since the copyright management is exercised depending on theindividual image qualities and applications, it is possible to balancethe demands for copyright protection from image providers with thedemands for use from users flexibly, instead of either-or choicesbetween permitting and prohibiting the use of images.

Up to this point, the present invention has been described inconjunction with several embodiments thereof. The foregoing embodimentshave been given solely by way of example. It will be understood by thoseskilled in the art that various modifications may be made tocombinations of the foregoing components and processes, and all suchmodifications are also intended to fall within the scope of the presentinvention.

Embodiment 8 has dealt with a hierarchical structure of frequencydivision type. However, a hierarchical structure of improvedapproximation accuracy type may be used instead. In that case, images ofvarious image quality levels can be decoded by discarding apredetermined number of lower order bits out of the bit string of thewavelet transform coefficients. That is, the copyright management unit2030 can adjust the image quality level by instructing the bit planedecoding unit 2256 of the number of bits to transform.

When this method of hierarchization is used, it is possible to make adistinction in image quality level between individual areas of an image.In that case, each of the profiles on the table for managing thecopyright level information shown in FIG. 17 may be described with threeparameters, i.e., application, image quality level, and image area, notthe two parameters or the application and image quality level alone. Theprofiles may also be described with two parameters of image qualitylevel and image area. Here, image providers can exercise more flexiblecopyright management than in the foregoing embodiment 8.

Different image qualities can be achieved not only by hierarchizing theresolution, but by hierarchizing the compression rate or the number ofcolors as well. The compression rate can be hierarchized by using thelayering function of JPEG 2000. Color images are expressed by luminancecomponents and color-difference components. Then, the color-differencecomponents may be subjected to the foregoing hierarchization techniques.

While the preferred embodiments of the present invention have beendescribed using specific terms, such description is for illustrativepurposes only, and it is to be understood that changes and variationsmay be made without departing from the spirit or scope of the appendedclaims.

1. An image display method comprising: displaying a moving image whilechanging a quality thereof with time in accordance with a predeterminedsetting value.
 2. The image display method according to claim 1, whereinthe setting value is one for reducing at least one of resolution, imagequality, and a frame rate of the moving image stepwise.
 3. The imagedisplay method according to claim 1, wherein the setting value is onefor reducing at least one of a luminance and color differences of themoving image stepwise.
 4. The image display method according to claim 1,wherein the setting value is one for narrowing down a display area ofthe moving image stepwise.
 5. The image display method according toclaim 4, wherein a predetermined advertisement is displayed outside adisplay area of the moving image.
 6. The image display method accordingto claim 1, wherein the setting value is fixed in response to apredetermined action of a user.
 7. An image coding apparatus comprising:a coding unit which codes moving image data; and an adding unit whichadds a predetermined setting value to the coded moving image data sothat the coded moving image data, when displayed, changes in qualitywith time.
 8. The image coding apparatus according to claim 7, whereinthe setting value is one for reducing at least one of resolution, imagequality, and a frame rate of the moving image stepwise.
 9. The imagecoding apparatus according to claim 7, wherein the setting value is onefor reducing at least one of a luminance and color differences of themoving image stepwise.
 10. The image coding apparatus according to claim7, wherein the setting value is one for narrowing down a display area ofthe moving image stepwise.
 11. An image coding apparatus comprising: acoding unit which codes moving image data so that the data, whendisplayed, changes in quality with time; and a quality setting unitwhich supplies the coding unit with a setting value for changing thequality.
 12. The image coding apparatus according to claim 11, whereinthe setting value is one for reducing at least one of resolution, imagequality, and a frame rate of the moving image stepwise.
 13. The imagecoding apparatus according to claim 11, wherein the setting value is onefor reducing at least one of a luminance and color differences of themoving image stepwise.
 14. The image coding apparatus according to claim11, wherein the setting value is one for narrowing down a display areaof the moving image stepwise.
 15. An image decoding apparatuscomprising: a decoding unit which decodes coded moving image data; and aquality setting unit which supplies the decoding unit with a settingvalue set so that quality changes with time, wherein the decoding unitdecodes the data while changing a quality thereof in accordance with thesetting value.
 16. An image decoding apparatus comprising: a recordingunit which records in advance basic data for reproducing contents ofpredetermined image data as a visible image; an acquisition unit whichacquires complementary data for complementing the basic data for thesake of improved image quality; an assembling unit which assembles thebasic data and the complementary data; and a decoding unit which decodesthe assembled image data.
 17. The image decoding apparatus according toclaim 16, further comprising a data request unit which requests data forcomplementing the basic data from a source of distribution after thebasic data is acquired, depending on processing capability of theapparatus.
 18. An image processing apparatus comprising: a decoding unitwhich acquires coded image data multiplexed so as to be decodable in aplurality of image qualities, and decodes the coded image data so as toproduce at least two or more images having different image qualities;and a memory unit which stores a plurality of pieces of image datadecoded in order to cope with predetermined processing.